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I. 


INTRODUCTION 


The  last  decade  has  seen  a  growing  interest  in  determining  the  impact  of  distance 
or  online  forms  of  education  on  grades  and  subsequent  job  performance.  Given  the  lower 
cost  of  delivering  such  degrees,  the  motivation  to  move  to  online  based  programs  has 
increased  substantially  in  recent  years.  While  these  cost  savings  are  certainly  relevant  in 
the  civilian  context,  the  military  recognizes  the  potential  for  additional  cost  savings  by 
not  having  to  relocate  personnel  for  the  sole  purpose  of  higher  education. 

This  popularity  and  cost  savings  have  led  to  much  academic  research  on  the 
subject.  Overall,  the  findings  on  the  impact  of  distance  or  online  educational  class 
formats  on  academic  performance  show  little  to  no  significant  difference  between  online 
and  traditional  programs.  Although,  existing  studies  do  indicate  that  synchronous 
distance  education  formats  are  inferior  to  asynchronous  formats  and  that  effects  tend  to 
be  heterogeneous  with  weaker  students  performing  proportionally  worse  in  distance 
formats.  However,  the  current  research  focuses  heavily  on  civilian  institutions.  This 
prompts  the  question:  Is  the  impact  of  distance  education  on  academic  and  job 
performance  outcomes  similar  at  military  institutions,  such  as  the  Naval  Postgraduate 
School  (NPS)? 

NPS  offers  several  online  degree  programs  that  come  under  the  broad  umbrella  of 
distance  learning  (DL)  programs.  The  Graduate  School  of  Operational  and  Information 
Sciences  (GSOIS)  offers  six  DL  master’s  programs  in  computer  science  related 
curriculums.  The  Graduate  School  of  Engineering  and  Applied  Sciences  (GSEAS)  offers 
eleven  DL  master’s  programs  in  engineering  related  curriculums.  The  Graduate  School  of 
Business  and  Public  Policy  (GSBPP)  offers  four  DL  master’s  programs  in  business  and 
contract  management  related  curriculums  (Naval  Postgraduate  School,  2016). 

In  this  thesis,  I  estimate  the  impact  of  DL  on  student  performance  and  subsequent 
labor  market  outcomes  for  NPS  military  students.  To  achieve  this  end,  I  merge  data  from 
three  sources — the  Institutional  Research,  Reporting,  and  Analysis  Office  at  NPS,  the 
Defense  Manpower  Data  Center  (DMDC),  and  the  Integrated  Postsecondary  Education 
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Data  System  (IPEDS) — to  create  a  large  sample  of  students  with  extensive  information 
on  their  demographics,  undergraduate  institution,  and  military  career  progression.  Using 
propensity  score  matching,  I  estimate  the  impact  of  DL  on  four  performance  outcomes: 
grade  point  average,  graduation,  promotion,  and  separation.  I  further  subdivide  the 
sample  into  various  subgroups  based  on  military  service  branch,  warfare  community, 
academic  preparation,  and  school  within  NPS  to  determine  the  heterogeneous  effects  of 
DL  within  each  subsample. 

My  findings  indicate  that  DL  students  at  NPS  on  average  have  lower  grades,  are 
less  likely  to  graduate,  are  less  likely  to  promote  in  their  subsequent  military  career,  and 
are  more  likely  to  separate  from  active  service.  Similar  to  the  literature  from  the  civilian 
sector,  I  also  find  heterogeneous  effects  of  DL.  However,  at  NPS  lower-ability  DL 
students  do  not  perform  worse  compared  to  higher-ability  DL  students. 

A.  SCOPE  OF  THIS  THESIS 

This  thesis  analyzes  NPS  students  in  both  DL  and  resident  programs  with 
academic  year  start  dates  between  2006  and  2013.  Data  for  the  years  in  question  and 
provided  by  the  Institutional  Research,  Reporting,  and  Analysis  Office  at  NPS  is  merged 
with  data  from  the  Defense  Manpower  Data  Center  (DMDC)  and  the  Integrated 
Postsecondary  Education  Data  System  (IPEDS)  to  supplement  the  range  of  students  with 
additional  demographic,  undergraduate  institution,  and  promotion  information.  The 
analysis  addresses  whether  or  not  there  is  a  difference  between  DL  and  resident  student 
performance  and  career  progression. 

This  research  is  quantitative  in  nature.  I  conduct  a  review  and  evaluation  of 
relevant  literature  with  a  focus  on  distance  education  and  online  programs  and  their 
perceived  impact  on  university  systems.  I  also  use  other  literature  to  provide  information 
relevant  to  my  research  topic. 

B.  RESEARCH  QUESTIONS 

1.  What  is  the  causal  impact  of  distance  learning  on  military  officer  students’ 
academic  and  job  performance? 
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2.  What  student  characteristics  best  predict  success  in  distance  learning  vs. 
resident  learning? 

3.  Are  there  systematic  differences  between  distance  learning  and  resident 
military  officer  demographic  characteristics,  MOS,  ability,  and/or 
academic  preparation? 

C.  ORGANIZATION  OF  THIS  THESIS 

This  thesis  consists  of  five  chapters.  Chapter  I  provides  a  brief  introduction  to  the 
subject  matter  and  methodology.  Chapter  II  presents  a  review  of  relevant  literature  with  a 
focus  on  the  impact  of  distance  education  on  performance  outcomes.  Chapter  III 
delineates  the  data  utilized  for  analysis,  demonstrates  difference  between  DL  and  resident 
student  bodies,  and  provides  an  in  depth  insight  into  the  methodology  behind  this  thesis. 
Chapter  IV  presents  and  explains  the  findings  of  this  thesis.  Finally,  Chapter  V  provides 
concluding  remarks  and  preliminary  recommendations. 
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II.  LITERATURE  REVIEW 


Given  the  recent  growth  in  distance  education  degree  programs  within  the  U.S. 
military  and  beyond,  there  is  an  increasing  need  to  better  understand  the  impact  of 
distance  education  on  student  learning  outcomes.  Hence,  the  large  literature  on  the  topic 
continues  to  grow  more  extensive  with  each  passing  year.  The  current  literature  falls  into 
three  categories:  meta-analyses,  observational  studies,  and  randomized  experiments.  I 
summarize  the  main  findings  from  these  categories  in  order  below. 

A.  META-ANALYSES 

Meta-analyses  are  a  quantitative  summary  of  existing  studies  and  present  a 
general  picture  of  the  state  of  research  in  a  particular  subject  area.  I  focus  on  three  key 
and  recent  studies  here  that  focus  on  how  online  and  distance  education  modes  of 
delivery  impact  outcomes.  A  U.S.  Department  of  Education  meta-analysis  conducted  by 
Means,  Toyama,  Murphy,  Bakia,  and  Jones  (2010)  looks  explicitly  at  online  education 
versus  traditional  face-to-face  instruction  focusing  on  post- secondary  education.  Lack 
(2013)  expounds  upon  Means  et  al.  (2010)  and  provides  additional  insight  for  future 
studies  in  this  area.  Unlike  these  studies,  Bernard  et  al.  (2004)  focus  explicitly  on 
distance  education,  separate  from  online  education,  and  discuss  at  length  the  issue  of 
synchronous  versus  asynchronous  delivery  methods.  Additionally,  there  are  a  variety  of 
outcomes  used  throughout  the  many  smaller  studies  included  in  these  meta-analyses, 
including  individual  course  grades,  overall  grade  point  average,  instructor  evaluation 
surveys,  and  student  evaluation  surveys.  However,  the  best  and  most  common  outcome 
chosen  is  individual  student  course  grade  because  it  allows  for  a  degree  of  control 
between  different  classes  and  instructors  and  bypasses  the  more  qualitative  nature  of 
surveys. 

Means  et  al.  (2010)  initially  set  out  to  provide  information  for  K-12  students. 
However,  the  vast  majority  of  existing  studies  revolve  around  secondary  and  post¬ 
secondary  education,  excluding  any  significant  measures  of  effect  for  the  intended 
student  body  (2010,  p.  31).  Also,  they  drop  a  large  number  of  studies  that  simply 
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compare  online  and  traditional  students,  as  they  are  likely  to  be  biased  by  a  student’s 
selection  into  a  particular  type  of  course.  By  screening  and  eliminating  all  but  45  from  a 
pool  of  1,132  existing  studies,  Means  et  al.  do  provide  several  significant  results  relevant 
to  continued  education  (2010,  p.  14).  After  combining  studies  and  weighting  results 
based  on  respective  sample  sizes,  they  find  that  the  online  delivery  method  is  just  as 
effective  as  face-to-face  instruction  but  not  any  better  (Means  et  al.,  2010,  p.  18).  Also, 
the  evidence  suggests  that  supplementing  classroom  instruction  with  online  resources, 
often  known  as  hybrid  delivery,  has  a  positive  and  significant  impact  on  student 
performance  as  measured  by  grades  (p.  19).  Further,  the  types  of  delivery  media  among 
online  courses  demonstrate  no  significant  effect  on  average  learning  outcomes  (Means  et 
al.,  2010,  p.  40).  Means  et  al.  (2010)  also  highlight  the  wide  variation  in  methods, 
findings,  and  effect  sizes  across  studies.  The  wide  variation  in  methodologies  likely 
contributes  to  the  wide  variation  in  findings.  Also,  although  they  eliminated  studies  based 
on  the  level  of  qualitative  analysis  and  chosen  outcome  variables  and  then  weighted 
based  on  sample  size,  many  studies  were  severely  biased  and  lacking  in  good  quality 
control  variables  (Means  et  al.,  2010,  p.  13). 

Lack  (2013)  conducts  a  meta-analysis  similar  to  that  of  Means  et  al.  (2010)  but 
with  several  key  differences  with  regard  to  her  focus.  Lack  (2013)  identifies  30  studies 
that  compare  some  combination  of  face-to-face,  online,  and/or  hybrid  learning.  She  also 
only  includes  studies  with  learning  outcomes,  such  a  course  grades,  as  dependent 
variables  and  precludes  studies  with  student  authors  (p.  8).  Instead  of  attempting  to 
calculate  overall  effects,  Lack  (2013)  asserts  that  the  existing  literature  is  inadequate  to 
determine  whether  or  not  online  or  hybrid  learning  modalities  are  more  or  less  effective 
than  traditional  face-to-face  modalities  (p.  10).  The  discussion,  in  turn,  revolves  around 
the  shortage  of  quality  studies,  small  sample  sizes,  and  need  for  random  assignment 
(Lack,  2013,  p.  8).  In  particular,  this  study  highlights  four  general  problems  with  the 
current  state  of  research.  First,  those  studies  that  achieve  random  assignment  to  distance 
education  have  very  small  sample  sizes.  Second,  those  studies  with  large  samples  often 
lack  sound  experimental  design  and  display  widely  conflicting  results.  Third,  existing 
observational  studies  generally  fail  to  account  for  self- selection  bias  despite  controlling 
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for  background  information.  And  finally,  the  remaining  studies  comprise  those  that 
neglect  relevant  controls  (Lack,  2013,  p.  11-12). 

Bernard  et  al.  (2004)  conduct  yet  another  meta-analysis,  although  older  than  the 
previous  two.  This  study  focuses  exclusively  on  distance  education,  which  can  be 
separate  from  online  education  that  does  not  involve  geographic  separation.  Findings 
include  a  small  yet  significant  positive  effect  of  distance  education  on  learning  outcomes 
(Bernard  et  al.,  2004,  p.  404).  However,  the  standard  errors  are  large,  and  thus  the  overall 
result  tends  to  be  more  in  accordance  with  Means  et  al.  (2010)  of  no  significant 
differences  in  student  outcomes  across  delivery  modes.  What  causes  this  study  to  stand 
out  is  the  emphasis  on  the  effects  of  synchronous  and  asynchronous  methods  of  distance 
education.  Synchronous  distance  education  results  in  a  significant  negative  effect  on 
learning  outcomes,  while  asynchronous  distance  education  results  in  a  significant  positive 
effect  on  learning  outcomes  (Bernard  et  al.,  2004,  p.  404).  Further,  all  three  outcomes 
measured — achievement,  attitude,  and  retention — yield  conflicting  results  between 
synchronous  and  asynchronous  formats.  With  respect  to  all  three  outcomes,  synchronous 
formats  favor  the  classroom  environment,  whereas  asynchronous  formats  favor  distance 
education  (Bernard  et  al.,  2004,  p.  408). 

These  large  meta-analyses  draw  on  a  vast  body  of  observational  studies  and  a 
smaller  body  of  randomized  studies.  I  discuss  each  of  these  in  turn  next.  Randomized 
experiments  account  for  selection  bias,  but  are  often  limited  in  other  areas  due  to 
difficulty  in  designing  and  implementing  proper  experiments  within  university  systems. 
Observational  studies  face  difficulty  correcting  for  selection  bias  or  simply  do  not  even 
attempt  to  account  for  it. 

B.  OBSERVATIONAL  STUDIES 

Koch  (2006)  conducted  an  interesting  observational  study  at  Old  Dominion 
University.  He  utilized  data  from  1994  to  2001  on  all  courses  with  both  a  distance 
learning  and  resident  component  (p.  24).  He  utilizes  the  resulting  sample  of  20,428 
observations  to  perform  OLS  regressions  of  student  characteristics  on  individual  course 
grades  (Koch,  2006,  p.  25).  Koch’s  (2006)  findings  indicate  that  distance  learning  has  a 
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very  small  impact,  if  any,  but  demographic  characteristics  do  significantly  impact  student 
grades  (p.26-28).  The  incredibly  large  sample  size  and  diverse  set  of  control  variables 
represent  this  study’s  greatest  strengths.  However,  these  strengths  may  not  be  enough  to 
balance  the  potential  biases  on  account  of  students  who  selected  into  the  residence  and 
distance  learning  courses.  That  said,  the  study  highlights  the  need  to  include  key 
demographic  variables  when  analyzing  the  impact  of  distance  education. 

Brown  and  Liedholm  (2002)  perform  another  observational  study,  but  take  a  more 
focused  approach  than  Koch  (2006).  They  attempt  to  compare  face-to-face  classroom, 
hybrid,  and  virtual — their  term  for  completely  online  class — versions  of  a  Principles  of 
Microeconomics  class.  All  three  versions  of  the  class  were  designed  to  be  very  similar 
and  utilize  similar  resources.  The  virtual  version  also  gained  access  to  recorded  lectures 
from  the  face-to-face  class  with  an  additional  synchronous  text  based  component  (Brown 
&  Liedholm,  2002,  p.  444).  Brown  and  Liedholm  (2002)  find  that  students  in  the  virtual 
version  of  class  consistently  score  approximately  half  a  letter  grade  below  students  in  the 
face-to-face  and  hybrid  classes  (p.  447).  Interestingly  enough,  they  designed  the  study 
and  acquired  demographic  data  as  a  control,  but  they  made  no  attempt  to  correct  for 
selection  bias  or  control  for  subtle,  yet  controllable,  difference  between  class  versions. 
For  example,  different  instructors  taught  the  face-to-face  and  hybrid  classes  (Brown  & 
Liedholm,  2002,  p.  444).  Another  issue  exists  in  the  small  and  disproportionate  sample 
size.  With  only  363  face-to-face,  258  hybrid,  and  89  virtual  student  observations,  the 
study  lacks  substantial  power  (Brown  &  Liedholm,  2002,  p.  445). 

Gratton-Lavoie  and  Stanley  (2009)  conduct  a  study  with  a  slightly  different 
methodology.  They  observe  eight  sections  of  an  introductory  economics  class  over  four 
semesters  to  compare  online  and  hybrid  delivery  methods  (p.  7).  Using  summary 
statistics  of  final  exam  scores,  Gratton-Lavoie  and  Stanley  (2009)  conclude  that  online 
education  has  a  positive  effect  on  learning  outcomes  (p.  12).  However,  this  method  is 
problematic  because  it  neglects  to  consider  potential  impacts  other  than  delivery  mode  on 
exam  score.  They  next  use  a  probit  model  to  estimate  the  likelihood  of  selecting  online 
over  hybrid.  From  these  likelihood  values,  they  further  assert  that  self-selection  bias 
shifts  the  positive  effect  found  earlier  to  an  insignificant  or  negative  one,  which  they  do 
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not  quantify  (Gratton-Lavoie  &  Stanley,  2009,  p.  20).  Despite  their  attempt  to  control  for 
selection,  their  small  sample  of  149  students  lacks  sufficient  power  to  estimate  the  effects 
of  online  and  hybrid  delivery  (p.  12).  Also,  as  mentioned  above,  utilizing  summary 
statistics  to  compare  average  final  exam  scores  between  online  and  hybrid  courses 
presents  serious  limitations  as  a  basis  for  determining  the  overall  effect  of  online 
education  on  learning  outcomes.  They  should  utilize  regression  analysis  when 
determining  effect  size  to  control  for  additional  factors  that  may  impact  learning 
outcomes,  such  as  previous  academic  performance  or  demographic  information. 

Coates,  Humphreys,  Kane,  and  Vachris  (2004)  perform  another  observational 
study  but  with  a  heavy  focus  on  correcting  for  any  self- selection.  They  obtain  data  from 
three  separate  universities  for  introductory  microeconomics  and  macroeconomics  courses 
(Coates  et  al.,  2004,  p.  535).  Coates  et  al.  (2004)  utilize  three  separate  models — OLS, 
2SLS,  and  an  endogenous  switching  equation — in  an  attempt  to  ascertain  the  influence  of 
selection  bias  on  Test  of  Understanding  College  Economics  (TUCE)  scores  between 
online  and  face-to-face  students.  They  find  that  selection  bias  presents  a  substantial  effect 
and  that  the  direction  of  the  bias  points  toward  zero  (Coates  et  al.,  2004,  p.  545).  The 
presence  of  effect  estimates  biased  toward  zero  when  ignoring  selection  sheds  additional 
light  on  the  multitude  of  observational  studies  with  findings  of  “no  significant  difference” 
between  online  and  face-to-face  modalities  (Coates  et  al.  2004,  p.  545).  Also  of  note, 
Coates  et  al.  (2004)  present  one  of  the  few  studies  to  address  systematic  differences 
between  students  self-selecting  online  courses  rather  than  face-to-face  offerings.  They 
find  that  students  selecting  online  or  distance  options  are  overwhelmingly  employed  full 
or  part  time  in  addition  to  school,  have  300  point  lower  SAT  scores  on  average,  and  tend 
to  perform  better  at  online  courses  compared  to  students  selecting  face-to-face  courses 
(Coates  et  al.,  2004,  p.  545).  Despite  the  lack  of  any  real  tangible  effect,  this  study  fills  a 
crucial  gap  in  the  literature  by  addressing  the  importance  of  handling  selection  bias  when 
comparing  student  performance  between  online  and  face-to-face  courses.  However,  as 
with  many  studies  on  the  topic,  Coates  et  al.  (2004)  have  only  a  very  small  sample. 
Samples  for  the  various  models  used  range  from  only  59  to  126  observations  (p.  537- 
544). 
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Olitsky  and  Cosgrove  (2012)  perform  a  unique  observational  study  attempting  to 
account  for  self- selection  bias.  They  compare  multiple  sections  of  Principles  of 
Microeconomics  and  Principles  of  Macroeconomics  courses,  designing  courses  with  a 
great  deal  of  control  between  blended  and  face-to-face  versions  (Olitsky  &  Cosgrove, 
2012,  p.  19).  Utilizing  exam  scores  as  outcome  variables,  they  also  find  no  significant 
difference  in  effects  between  blended  and  face-to-face  versions  (Olitsky  &  Cosgrove, 
2012,  p.  30).  This  study  stands  out  for  its  method  of  correcting  for  self-selection  bias. 
Olitsky  and  Cosgrove  (2012)  determine  the  average  treatment  effect  of  blended  learning 
by  utilizing  propensity  score  matching  to  create  a  matched  sample,  mitigating  the 
potential  for  selection  bias  (p.  27).  It  is  this  same  method  that  we  employ  in  our  study  to 
correct  for  selection  bias.  As  such,  Olitsky  and  Cosgrove  (2012)  provide  a  clear 
precedent  for  the  use  of  propensity  score  matching  to  correct  for  self- selection  bias  when 
unable  to  randomly  assign  students  between  treatment  and  control  groups. 

However,  despite  such  methods  of  correction  for  selection  bias  among 
observational  experiments,  random  assignment  of  students  provides  the  only  true  means 
of  completely  eliminating  such  bias.  Consequently,  the  following  randomized 
experiments  represent  some  of  the  best  attempts  to  determine  the  effect  of  distance 
education  within  the  current  field  of  study. 

C.  RANDOMIZED  STUDIES 

Harmon,  Alpert,  and  Lambrinos  (2014)  design  an  experiment  to  emulate  random 
assignment  of  students.  They  randomly  divide  a  Principles  of  Economics  class  into 
various  portions  of  online  or  face-to-face  delivery  based  on  chapters.  They  then  compute 
the  likelihood  of  answering  midterm  and  final  exam  questions  correctly  based  on  whether 
the  associated  questions  correspond  to  online  or  face-to-face  portions  of  the  course  using 
a  logit  model  (Harmon  et  al.,  2014,  p.  116-118).  Harmon  et  al.  (2014)  reiterate  the  “no 
significant  difference”  (p.  118)  findings  of  several  other  studies,  yet  their  approach 
remains  unique.  However,  this  uniqueness  coupled  with  a  very  small  sample  of  36 
students  leads  to  several  issues.  The  article  lacks  clarity  on  many  details  and  leaves  the 
reader  unsure  as  to  the  validity  of  the  approach.  For  example,  Harmon  et  al.  (2014)  make 
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no  mention  of  the  number  of  sections,  and  based  on  sample  size  a  single  section  is  likely. 
This  implies  that  particular  exam  questions  were  only  either  observed  as  online  or  face- 
to-face  when  computing  estimates.  The  resulting  lack  of  control  among  outcome 
variables  suggests  the  potential  for  serious  bias.  Harmon  et  al.  (2014)  may  eliminate  self¬ 
selection  bias  from  their  sample,  but  shortcomings  elsewhere  leave  much  to  be  desired. 

Bowen,  Chingos,  Lack,  and  Nygren  (2014)  design  a  randomized  experiment 
spanning  six  different  universities.  They  compare  traditional  and  hybrid  formats  for  a 
statistics  course  and  conclude  that  hybrid  formats  offer  a  “no-harm-done”  alternative  that 
also  results  in  less  time — both  for  instruction  and  completion  of  deliverables — for 
students  (Bowen  et  al.,  2014,  p.  107).  This  otherwise  well  designed  experiment  does 
suffer  from  a  lack  of  control  among  instructors,  which  is  openly  presented  to  the  reader 
(Bowen  et  al.,  2014,  p.  101).  Also,  this  study  uses  the  largest  sample  among  randomized 
experiments  with  605  participants  (Bowen  et  al.,  2014,  p.  98). 

Perhaps  the  most  widely  cited  study  within  the  field,  Figlio,  Rush,  and  Yin  (2010) 
conduct  a  randomized  experiment  comparing  exam  scores  between  online  and  live 
versions  of  a  Principles  of  Microeconomics  course  (p.  766).  Figlio  et  al.  (2010)  offer 
students  extra  credit  points  in  exchange  for  participating  in  an  experiment.  They  then 
randomly  assign  the  327  students  agreeing  to  participate  to  either  live  or  online  versions 
of  the  class  (Figlio  et  al.,  2010,  p.  767).  Figlio  et  al.  (2010)  find  no  statistical  difference  in 
outcomes  between  online  and  live  versions  of  the  class  (p.  779).  Also  of  note,  the  nature 
of  the  experiment  makes  comparison  to  distance  learning  difficult.  For  example,  students 
in  the  online  section  still  had  access  to  instructor  office  hours  and  could  schedule 
individual  face-to-face  meetings  with  the  instructor  (Figlio  et  al.,  2010,  p.  766). 

Joyce,  Crockett,  Jaeger,  Altindag,  and  O’Connell  (2014)  conduct  another 

randomized  experiment  comparing  traditional  and  hybrid  formats.  They  also  offer  extra 

credit  points  in  exchange  for  participation.  They  observe  the  656  participants  spread 

through  eight  sections  of  a  Principles  of  Microeconomics  course,  and  outcome  variables 

include  midterm  and  final  exams,  Applia  coursework,  course  final  grades,  and 

withdrawals  (Joyce  et  al.,  2014,  p.  6-9).  Joyce  et  al.  (2014)  find  that  “traditional  does 

moderately  better”  (p.  27),  and  on  average  students  in  traditional  versions  perform 
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2.5  percentage  points  higher  on  exams  than  those  in  hybrid  formats  (p.  28).  Despite  the 
seemingly  common  small  sample  size,  this  study  performs  exceptionally  well.  Joyce  et  al. 
(2014)  manage  to  control  for  a  multitude  of  factors  such  as  minimizing  differences 
between  classes  and  resources  and  including  a  large  set  of  demographic  and  academic 
performance  variables. 

Alpert,  Couch,  and  Harmon  (2015)  perform  one  of  the  best  randomized  studies  so 
far.  They  randomly  assign  participants,  who  again  were  offered  extra  credit  for 
participation,  into  either  face-to-face,  online,  or  hybrid  sections  of  a  microeconomics 
principles  class  between  Fall  of  2012  and  Spring  of  2014  (Alpert  et  al.,  2015,  p.  3-4). 
Alpert  et  al.  (2015)  utilize  three  stages  of  OLS  regressions — each  stage  increasing  the 
number  of  controls — to  determine  the  effects  of  the  three  class  formats  on  cumulative 
final  exams  scores  (p.  4).  They  find  that  blended  format  never  yields  a  significant  effect, 
but  online  classes  result  in  a  significant  and  consistent  decrease  of  approximately  half  a 
letter  grade  (Alpert  et  al.,  2015,  p.  27).  The  only  concern  with  this  study  again  comes  in 
the  form  of  a  relatively  small  sample  size,  but  this  seems  to  be  a  pervasive  issue  among 
randomized  experiments. 

The  existing  literature  presents  several  relevant  methodologies  as  well  as  many 
concerns.  No  studies  exist  addressing  the  issue  of  online  or  distance  education  compared 
to  resident  education  for  military  officer  students.  In  designing  my  study  I  attempt  to 
apply  the  lessons  presented  above  to  a  sample  of  military  officer  students  at  NPS. 
Unfortunately,  I  am  unable  to  achieve  the  gold  standard  of  conducting  a  randomized 
experiment.  However,  I  instead  focus  on  a  methodology  derived  from  both  Koch  (2006) 
and  Olitsky  and  Cosgrove  (2012).  Because  Koch  (2006)  found  a  large  impact  of 
demographics  on  performance  outcomes,  I  focus  on  a  large  sample  with  robust 
demographic  controls.  Also,  I  utilize  propensity  score  matching,  similar  to  Olitsky  and 
Cosgrove  (2012),  to  mitigate  biases  resulting  from  students  self-selecting  into  either 
distance  or  resident  delivery  modes. 
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III.  DATA/METHODOLOGY 


Utilizing  several  of  the  strengths  spread  across  studies  within  the  existing 
literature,  I  design  an  observational  study  to  estimate  the  impact  of  DL  on  learning  and 
military  performance  outcomes.  I  merge  data  from  three  separate  sources — the  Naval 
Postgraduate  School  (NPS),  the  Defense  Manpower  Data  Center  (DMDC),  and  the 
Integrated  Postsecondary  Education  Data  System  (IPEDS) — to  create  a  sample 
containing  the  necessary  demographic  controls.  I  then  use  propensity  score  matching  to 
correct  for  bias  resulting  from  selection  into  either  DL  or  residency  programs.  Finally,  I 
conduct  regression  analysis  to  determine  the  impact  of  DL  on  various  academic  and 
military  performance  related  dependent  variables.  This  section  addresses  each  of  these 
data  sources  and  methodologies  in  turn. 

A.  NPS  DATA 

1.  Sample 

The  primary  data  come  from  a  Python  extract  provided  by  the  Institutional 
Research,  Reporting,  and  Analysis  Office  at  NPS.  These  data  were  used  in  a  master’s 
thesis  by  Kyle  Alcock  at  NPS  in  March  2015,  but  that  study  focused  only  on  a  sample  of 
Naval  officers.  The  subjects  in  my  study  initially  consist  of  the  population  of  10,882  NPS 
students — U.S.  military  officers,  civilians,  and  international  students — who  began 
academic  programs  between  the  2006  and  2013  academic  years.  Civilian  and 
international  students  are  excluded  from  the  analysis  sample,  as  I  cannot  match  DMDC 
information  to  them.  After  dropping  all  civilian  and  international  students  the  sample  size 
decreases  to  6,754  observations. 

2.  Independent  Variables 

a.  Treatment  Indicator 

The  NPS  data  provide  the  binary  variable  for  DL,  where  DL  equals  one  for 
students  enrolled  in  NPS  DL  degree  programs.  DL  is  the  indicator  of  treatment  for  this 
study. 
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b.  NPS  Institutional  Controls 

I  generate  cohort  control  variables  from  academic  start  year  and  quarter 
information.  Also,  school  indicators  allow  control  for  the  four  different  schools  at  NPS: 
the  Graduate  Schools  of  Business  and  Public  Policy  (GSBPP),  the  Graduate  School  of 
Engineering  and  Applied  Sciences  (GSEAS),  the  Graduate  School  of  Operational  and 
Information  Sciences  (GSOIS),  and  the  School  of  International  Graduate  Studies  (SIGS). 

c.  Academic  Preparation 

NPS  utilizes  academic  profile  codes  (APC)  as  the  primary  means  for  determining 
academic  eligibility  and  student  academic  preparation  prior  to  admission  at  NPS.  A 
student’s  APC  consists  of  three  discrete  digits,  each  representing  a  different  aspect  of  his 
or  her  academic  background.  The  first  digit  represents  undergraduate  academic 
performance  based  on  GPA,  the  second  digit  represents  mathematics  background  and/or 
elapsed  time  since  college  level  math  course  completion,  and  the  third  digit  represents 
engineering,  science,  or  technical  background  (Naval  Postgraduate  School,  2016).  To 
control  for  student  preparation  I  generate  dummy  variables  for  each  digit  of  APC 
indicating  whether  each  student  has  met  the  requisite  APC  for  his  or  her  particular 
curriculum. 

Also,  the  NPS  data  include  information  on  both  undergraduate  school  name  and 
time  elapsed  from  completion  of  an  undergraduate  degree  prior  to  beginning  studies  at 
NPS.  Undergraduate  school  name  allows  for  the  merger  with  IPEDS  data  from  the 
National  Center  of  Education  Statistics  (NCES)  that  indicate  the  sector — private  vs. 
public  and  for-profit  vs.  not-for-profit — of  the  college  of  university  the  military  officer 
graduated  from.  In  addition,  I  use  undergraduate  school  name  to  generate  an  indicator  of 
whether  the  student  attended  a  service  academy.  With  a  completely  military  sample, 
attending  a  service  academy  also  represents  a  level  of  academic  preparation.  Finally, 
because  there  is  variation  in  elapsed  time  between  undergraduate  and  graduate  educations 
among  NPS  DL  and  resident  students,  the  inclusion  of  the  continuous  variables  for  years 
since  undergraduate  education  seems  relevant  as  yet  another  measure  of  academic 
preparation. 
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d.  Service  and  Community 

The  data  provide  information  on  NPS  students’  branch  of  military  service  as  well 
as  designator.  Designator  or  Military  Occupational  Specialty  (MOS)  represents  the 
particular  job  a  service  member  performs.  From  this  information,  I  generate  several 
community  variables  and  their  respective  interactions  with  service  branch.  Table  1 
displays  the  communities  and  respective  designator/MOS  by  service.  I  generate  dummy 
variables  for  each  service,  community,  and  the  interactions  of  each  service  and 
community  as  controls  in  the  analysis  performed  in  Chapter  IV. 

Table  1.  Service  and  community  breakdown 


Navy 

Army 

Air  Force 

Job 

Designator 

Approx,  n 

Job 

MOS 

Approx,  n 

Job 

MOS 

Approx,  n 

814 

0 

o 

Surface 

Surface 

1110 

786 

Surf.  Undes. 

1160 

28 

■ 

0 

0 

Submarine 

Submarine 

1120 

302 

Sub.  Undes. 

1170 

45 

m  giw 

47 

162 

General 

1300 

10 

Pilot 

75xx 

146 

Aviation 

15xx 

47 

Pilot 

llxx 

126 

Aviation 

Pilot 

1310 

729 

NFO 

7525/7528 

33 

Combat  Sys.  Off. 

12xx 

36 

Pilot  Undes. 

1390 

21 

NFO 

1320 

279 

NFO  Undes. 

1370 

3 

Wi/'  ~ 

71 

88 

Oceanography 

1800 

93 

MAGTF 

02xx 

29 

Intel 

35xx 

71 

Intel 

14xx 

72 

IWO 

1810 

83 

Ground 

02xx 

7 

Cyber 

17xx 

16 

Intelligence 

IPO 

1820 

66 

CI/HUMINT 

02xx 

4 

Intel 

1830 

75 

SI/EWO 

02xx 

12 

Air 

02xx 

14 

CI/HUMINT  OPS 

02xx 

3 

453 

0 

Spec.  War 

1130 

46 

Infantry 

03xx 

51 

Infantry 

llxx 

100 

EOD 

1140 

29 

Tank  Off. 

18xx 

4 

Armor 

19xx 

2 

around  comoat 

EOD  Undes. 

1190 

1 

AAV  Off. 

18xx 

2 

Field  Artillery 

13xx 

87 

Field  Artillery  Off. 

08xx 

38 

A/D  Artillery 

14xx 

17 

Special  Forces 

18xx 

247 

294 

502 

EDO 

1440 

11 

Adjutant 

Olxx 

39 

Imm.  8i  Personnel 

OOxx 

29 

Weather 

lSxx 

71 

EDO  Undes. 

1460 

281 

Logistics  Off. 

04xx 

81 

Corps,  of  Eng. 

12xx 

68 

Ops.  Support 

16xx 

42 

AEDO  Aero 

1510 

44 

Comms.  Off. 

06xx 

111 

Signal  Corps. 

25xx 

52 

Security  Forces 

31xx 

19 

AEDO  Maint 

1520 

78 

Combat  Eng.  Off. 

13xx 

25 

MP 

31xx 

7 

Civil  Engineering 

32xx 

16 

PAO 

1600 

174 

Ground  Supp.  Off. 

30xx 

66 

Psy.  Ops. 

37/39xx 

5 

Public  Affairs 

35xx 

10 

FAO 

1710 

13 

Finance  Off. 

34xx 

32 

Civil  Affairs 

38xx 

27 

Personnel 

38xx 

12 

FAO  Undes. 

1720 

42 

PAO 

43xx 

3 

Space  Ops. 

40xx 

1 

Biomedical  Sp 

43xx 

2 

HR 

1200 

189 

JAG 

44zz 

6 

Adjutant  Gen. 

42xx 

10 

Scientific  Research 

61xx 

11 

Medical 

2100 

10 

MP 

58xx 

5 

Finance 

44x 

7 

Developmental  Eng. 

62xx 

44 

Dental 

2200 

3 

A/C  Maint.  Off. 

60xx 

22 

Foreign  Affairs 

48xx 

8 

Acquisitions 

63xx 

61 

Medical  Service 

2300 

65 

Avioincs  Off. 

63xx 

1 

Acquisitions  Off. 

51xx 

8 

Contracting 

64xx 

79 

Flight  Surgeon 

2302 

1 

Avn.  Supply  Off. 

66xx 

27 

Medical 

60xx 

1 

Finance 

65xx 

24 

Support 

JAG 

2500 

7 

Air  C8tC  Off. 

72xx 

9 

Medical  Service 

67xx 

9 

Special  Inv. 

71xx 

53 

Nurse 

2900 

12 

Low  Alt.  A/D 

72xx 

9 

T  ransportation/Logistics 

88xx 

14 

A/C  Maint. 

21xx 

17 

Supply 

3100 

332 

Air  Supp.  Control 

72xx 

11 

Ammunition/Maint/Ord. 

91xx 

20 

Missile  Maint. 

21xx 

2 

Chaplain 

4100 

2 

A/D  Control 

72xx 

8 

Quartermaster 

92xx 

16 

Readiness 

21xx 

24 

Civil  Engineering  Corps 

5100 

20 

ATC 

72xx 

10 

O.R./Sys  Analysis 

49/57xx 

4 

Spec.  Duty 

88xx 

2 

Spec.  WarLDO 

6152 

1 

Avn.  Acquisitions 

80xx 

1 

Chemical 

74xx 

8 

Students/candidates 

92xx 

13 

Elec.  Surf.  LDO 

6180 

_ 1 

Defense  Systems  Analyst 

88xx 

1 

Eng.  Sub  LDO 

6232 

1 

Elec.  Sub  LDO 

6280 

1 

Comm.  Sub  LDO 

6290 

1 

Avn.  Maint.  LDO 

6330 

6 

Air  Traffic  Control  LDO 

6390 

1 

Admin  LDO 

6410 

5 

IS  LDO 

6420 

2 

Met/Ocean  LDO 

6460 

1 
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3.  Dependent  Variables 

The  NPS  data  provide  three  of  the  five  dependent  variables  that  I  analyze.  Total 
quality  point  rating  (TQPR)  is  essentially  a  GPA  calculated  for  all  courses  taken  at  NPS 
(Naval  Postgraduate  School,  2015).  I  use  TQPR  as  the  primary  and  only  continuous 
outcome  variable.  It  would  be  advantageous  to  utilize  individual  course  grades  instead  of 
overall  TQPR,  but  this  information  is  currently  unavailable  for  the  existing  sample.  I  also 
include  binary  outcome  of  whether  a  student  graduates. 

B.  DMDC  DATA 

1.  Sample 

As  Koch  (2006)  makes  clear,  sound  demographic  control  variables  are  necessary 
when  ascertaining  the  impact  of  DL.  DMDC  data  provide  these  important  demographic 
controls  for  the  sample,  as  well  as  subsequent  career  progression  of  these  students  upon 
leaving  NPS.  Demographic  data  were  requested  for  the  entire  sample  as  of  six  months 
prior  to  beginning  their  studies  at  NPS,  while  work  performance  data  was  requested 
covering  the  period  after  leaving  NPS.  Thus,  the  sample  size  remains  the  same,  and  forty- 
seven  additional  observables  are  added. 

2.  Independent  Variables 

Of  the  additional  observable  characteristics  added,  I  focus  on  key  demographic 
variables  including:  gender,  race,  marital  status,  age,  number  of  dependents,  and  rank. 
Race  consists  of  variables  for  white,  black,  Hispanic,  and  other  race.  Marital  status 
includes  two  dummy  variables,  one  indicating  whether  an  observation  was  married 
during  his  or  her  time  at  NPS  and  the  other  indicates  if  the  observations  had  ever 
experienced  a  divorce.  Rank  indicates  the  military  paygrade  of  an  observation  upon 
beginning  enrollment  at  NPS.  The  fourteen  resulting  variables  cover  the  range  of 
demographics  typically  controlled  for. 

3.  Dependent  Variables 

DMDC  data  provide  the  remaining  two  outcome  variables  in  this  study,  which  are 

promotion  and  separation  status.  The  dependent  variable  “promoted”  indicates  students 

16 


who  received  a  promotion  after  departure  from  NPS,  whether  or  not  they  had  actually 
received  a  degree.  Separated  indicates  those  officers  that  separated  from  military  service 
for  any  reason  after  leaving  NPS. 

C.  IPEDS  DATA 

1.  Sample 

IPEDS  is  a  branch  of  the  National  Center  for  Education  Statistics  (NCES)  and 
provides  downloadable  and  publically  available  data  for  accredited  schools  offering 
postsecondary  education  within  the  United  States  (National  Center  for  Education 
Statistics,  2016a).  I  merge  the  2012  IPEDS  database,  which  is  the  most  applicable,  to  the 
existing  sample  based  on  undergraduate  degree  institution  listed  in  the  NPS  data. 

2.  Variables 

Unlike  the  several  variables  DMDC  data  provide,  I  only  use  IPEDS  for  the 
addition  of  the  single  independent  variable  “sector.”  The  IPEDS  sector  represents  a  scale 
of  one  through  nine  accounting  for  both  control  and  level  of  an  institution.  Control 
represents  whether  the  school  is  public,  private  not-for-profit,  or  private  for-profit.  The 
level  indicates  whether  a  school  offers  four,  two,  or  less-than-two-year  degrees  (National 
Center  for  Education  Statistics,  2016b).  Table  2  illustrates  the  various  values  for  sector  as 
provided  by  IPEDS  (National  Center  for  Education  Statistics,  2016c). 


Table  2.  IPEDS  sector  breakdown 


Code 

Definition 

1 

Public,  4-year  or  above 

2 

Private  not-for-profit,  4-year  or  above 

3 

Private  for-profit,  4-year  or  above 

4 

Public,  2-year 

5 

Private  not-for-profit,  2-year 

6 

Private  for-profit,  2-year 

7 

Public,  less  than  2-year 

8 

Private  not-for-profit,  less  than  2-year 

9 

Private  for-profit,  less  than  2-year 
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However,  I  focus  solely  on  the  control  classification  because  commissioned 
officers  overwhelmingly  possess  four  year  degrees.  As  such,  I  generate  three  control 
variables  for  the  entire  sample:  public,  private  not-for-profit,  and  private  for-profit. 
Additionally,  I  define  these  three  variables  so  that  they  are  exclusive  of  Service  Academy 
graduates.  Service  Academies  fall  under  public  category  according  to  IPEDS,  however 
based  on  selectivity  and  average  student  performance  prior  to  postsecondary  education 
they  are  more  akin  to  private  not-for-profit  institutions.  For  this  reason,  and  because  a 
large  proportion  of  NPS  students  are  Service  Academy  graduates,  I  utilize  the  variable 
“Service  Academy”  from  the  NPS  data  to  represent  a  fourth  component  of  sector. 

D.  DATA  SUMMARY 

The  final  merged  sample  retains  the  original  6,754  observations,  approximately 
20  percent  of  which  are  for  DL  students.  Table  3  presents  a  summary  of  the  number  of 
observations  within  various  subgroups  of  the  sample. 
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Table  3.  Sample  summary 


DL 

Resident 

All 

Full  Sample 

1,331 

5,423 

6,754 

By  Variable: 

u  Navy 

1,182 

2,773 

3,955 

Marine 

54 

767 

821 

$  Army 

16 

910 

926 

Air  Force 

69 

887 

956 

Surface 

104 

710 

814 

Submarine 

122 

225 

347 

3  Aviation 

697 

764 

1,461 

E  Intelligence 

24 

690 

714 

o 

U  Ground  Combat 

33 

776 

809 

Support 

337 

2,449 

2,786 

01 

29 

122 

151 

02 

46 

473 

519 

%  03 

706 

2,883 

3,589 

c2  04 

272 

1,726 

1,998 

05 

130 

50 

180 

06 

14 

4 

18 

Public 

457 

2,461 

2,918 

S  Private  not-for-profit 
c/3  Private  for-profit 

223 

1,108 

1,331 

5 

41 

50 

Service  Academy 

412 

1,230 

1,642 

GSBPP 

772 

1,150 

1,922 

§  GSEAS 

361 

1,055 

1,416 

§  GSOIS 

190 

1,697 

1,887 

SIGS 

8 

1,384 

1,392 

Met  APC  1 

1,019 

4,863 

5,882 

^  Met  APC  2 

845 

4,166 

5,011 

Met  APC  3 

1,084 

4,887 

5,971 

Female 

70 

523 

593 

8  White 

949 

4,021 

4,970 

&  Black 

M  Hispanic 

8  Other  Race 

Q  Married 

64 

356 

420 

82 

375 

457 

236 

671 

907 

961 

3,963 

4,924 

Divorced 

4 

102 

106 
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There  are  several  systematic  differences  between  DL  and  resident  students. 
Figures  1  through  1 1  illustrate  the  differences  between  DL  and  resident  students  based  on 
the  independent  variable  categories  of  service,  community,  rank,  school,  APC, 
undergraduate  degree  institution  sector,  gender,  marital  status,  and  race. 

A  comparison  of  services  between  DL  and  residency  are  depicted  in  Figure  1. 
Unsurprisingly,  Navy  students  comprise  the  majority  among  residents.  However,  for  DL 
this  majority  nearly  doubles.  Marine  Corps,  Army,  and  Air  Force  students  comprise  only 
about  10  percent  of  all  DL  students  at  NPS  between  2006  and  2013. 


Figure  1.  Military  service  branch  comparison  for  DL  and  residency 


Warfare  communities  are  depicted  in  Figure  2.  The  makeup  of  DL  and  resident 
students  is  quite  different.  In  particular,  aviators  tend  to  favor  and/or  get  assigned  to  DL 
programs  more  than  other  communities.  While  aviators  only  make  up  approximately  12 
percent  of  all  resident  students  between  2006  and  2013,  they  comprise  more  than  half  of 
the  DL  students  for  the  same  period.  The  disparities  presented  only  reinforce  the  need  to 
control  for  both  service  and  community  in  assessing  student  outcomes,  and  also  in  order 
to  make  an  apples-to-apples  comparison. 
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Figure  2.  Warfare  community  comparison  for  DL  and  residency 


Intelligence 

2% 


Ground 

Combat 


DL  Surface 


Submarine 


Resident 


Figure  3  illustrates  rank  breakdowns  between  DL  and  residency.  The  only 
substantial  differences  seem  to  be  a  decrease  in  0-2’s,  a  decrease  in  0-4’s,  and  an 
increase  in  0-5’ s  within  DL.  Figure  4  shows  the  increased  representation  of  GSBPP 
within  DL  compared  to  residency,  while  also  displaying  a  decreasing  representation  by 
GSOIS  and  SIGS. 

Figure  3.  Military  rank  comparison  for  DL  and  residency 

Resident 
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Figure  4.  NPS  school  comparison  for  DL  and  residency 


DL 

SIGS 


Resident 


Figures  5  through  7  depict  whether  or  not  students  met  the  required  APCs  for 
their  respective  curriculums.  Interestingly  enough,  comparing  APC  results  show  that  DL 
students  tend  to  be  far  less  prepared  with  regard  to  all  APC’s  than  resident  students.  The 
greatest  gap  is  in  APC  2  (mathematics  background). 


Figure  5.  APC  1  undergraduate  GPA  comparison  for  DL  and  residency 

DL  Resident 
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Figure  6.  APC  2  undergraduate  mathematics  background  comparison  for  DL 

and  residency 


DL 


Resident 


Figure  7.  APC  3  undergraduate  science  and  technical  background 
comparison  for  DL  and  residency 


DL 


Resident 


Figure  8  depicts  the  IPEDS  sector  breakdown.  Sector  distribution  in  DL  is 
comparable  with  the  distribution  in  residency  programs,  with  no  difference  in  overall 
percentage  of  private  not-for-profit  and  private  for-profit  institutions.  However,  students 
with  undergraduate  degrees  from  public  institutions  decrease  by  10  percentage  points 
within  DL  compared  to  residency,  while  an  increase  in  Service  Academy  graduates 
makes  up  the  difference.  Gender,  shown  in  Ligure  9,  also  shows  only  a  small  difference 
between  DL  and  residency.  There  are  4  percentage  points  fewer  females  in  DL  than  in 
residency. 
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Figure  8.  Undergraduate  institution  sector  comparison  for  DL  and  residency 


DL 


Resident 


Figure  9.  Gender  comparison  for  DL  and  residency 


DL  and  resident  military  students  tend  to  be  similarly  distributed  with  regard  to 
marital  status  as  well.  Figure  10  shows  that  DL  students  represent  only  1  percentage  point 
more  unmarried  students  than  their  resident  counterparts. 


24 


Figure  10.  Marital  status  comparison  for  DL  and  residency. 


DL 


Resident 


Figure  11  depicts  the  race  breakdown  between  DL  and  residency.  Race  presents 
yet  another  similar  distribution  between  DL  and  residency,  with  only  a  slight  increase  in 
other  race  within  DL. 


Figure  11.  Race  comparison  for  DL  and  residency. 


DL 

Other/Unknown 


Resident 


Clearly,  there  exist  systematic  differences  between  DL  and  resident  students  at 
NPS.  Students  who  are  in  the  Navy  and  in  the  aviation  community  tend  to  be  over¬ 
represented  in  DL.  Also,  DL  students  suffer  from  being  less  prepared  on  average, 
especially  in  mathematics,  than  resident  students.  These  are  but  a  few  of  the  large 
differences  between  DL  and  residency,  yet  many  subtle  disparities  also  exist  in  the  data. 
Thus,  it  is  imperative  to  include  all  of  the  above  categories  as  controls  and  to  use  them  for 
propensity  score  matching. 
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E.  METHODOLOGY 


The  method  of  propensity  score  matching  conceptually  involves  creating 
counterfactual  outcomes  of  what  would  have  happened  to  DL  students  had  they  gone 
through  a  resident  program,  and  what  would  have  happened  to  resident  students  had  they 
been  through  a  DL  program. 

The  matching  methodology  is  broken  down  into  two  distinct  stages.  Stage  One 
involves  creating  a  matched  subsample  from  the  existing  sample.  Stage  Two  of  my 
analysis  then  uses  the  matched  sample  to  conduct  regression  analysis  and  determine  the 
impact  of  DL  on  performance  outcomes.  I  further  discuss  the  methodology  behind  these 
two  stages  in  the  next  two  sections. 

1.  Stage  One 

Propensity  score  matching  is  a  statistical  technique  that  allows  for  the  creation  of 
a  matched  sample  where  observations  are  similar  enough  between  treatment  and  control 
groups  to  efficiently  and  unbiasedly  determine  the  average  effect  of  the  treatment.  Also, 
on  can  determine  the  treatment  effect  on  the  treated  using  this  method.  Matching  relies  on 
the  key  assumption  that  subjects  that  are  similar  based  upon  observable  characteristics  are 
likely  also  similar  on  unobservable  characteristics  (Gertler,  2011).  In  this  study,  the 
variable  DL  represents  treatment.  Consequently,  within  a  matched  sample,  those 
observations  enrolled  in  DL  are  similar  with  respect  to  observable  characteristics,  and 
thus  also  unobservable  characteristics,  to  those  enrolled  as  resident  students.  In  a  sense, 
this  process  is  simulating  a  nonexistent  control  group  as  a  counterfactual  for  the  purposes 
of  examining  the  effect  of  DL  on  learning  and  job  performance  outcomes.  Thus,  the 
counterfactual  match  to  a  DL  student  is  identified  by  finding  a  control  resident  with  the 
same  propensity  to  be  in  DL  and  vice  versa. 

To  create  this  matched  sample,  I  start  by  determining  the  probability  that  each 
observation  is  enrolled  in  DL.  Considering  DL  is  a  binary  variable,  a  Logit  model  suits 
this  purpose  well.  Equation  1  specifies  this  logit  model,  where  the  associated  independent 
variables  are  defined  in  Table  4. 
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prob{  DL,  =  1)  = 


^bQ+biDEMOj+b2ACADi+b}MILi+b4CHRTj 
j  ^b0+blDEMOi+b1ACADi+b,MILi+biCHRTi 


(1) 


Table  4.  Independent  variables 


Demographic  Controls  (DEMO) 

Academic  Controls  (ACAD) 

Military  Controls  (MIL) 

Cohort  Controls  (CHRT) 

Female 

Years  since  Undergraduate  Degree  (Continuous) 

Service  (Navy  Reference) 

School  (GSBPP  Referece) 

Race  (White  Reference) 

Sector  (Public  Reference) 

Community  (Support  Reference) 

Cohort  Year  (2006  Reference) 

Married 

Met  APC  1,2,3 

Service  Community  Interactions 

Cohort  Quarter  (Q 1  Reference) 

Divorced 

Rank  (0-3  Reference) 

Age  (Continuous) 

Number  of  Dependents  (Continuous) 

Upon  estimating  this  regression,  predicted  values  of  DL  are  calculated  for  all 
observations.  I  then  utilized  a  kernel  density  function  to  plot  probability  density  overlays 
for  both  DL=1  and  DL=0.  This  overlay  allows  for  visual  inspection  of  the  region  of 
common  support,  or  the  region  where  observations  are  similar  in  their  propensity  for 
treatment  (Gertler,  2011).  Ideally  the  probability  distributions  do  not  overlap  perfectly,  as 
the  goal  is  to  eliminate  dissimilar  observations  at  the  far  left  and  right  ends  of  the  overlay. 
Figure  12  presents  an  example  overlay  plot. 

Figure  12.  Example  propensity  score  matching  overlay 


Source:  Gertler,  2011,  p.  110 


27 


To  achieve  a  sound  probability  overlay  requires  some  trial  and  error.  I  modified 
the  model  in  Equation  1  several  times  to  acquire  a  desirable  overlay  and  satisfy  the 
necessary  assumption  of  common  support.  I  eventually  acquire  a  suitable  region  of 
common  support  by  attempting  various  combinations  of  independent  variables.  I  then 
create  the  final  matched  sample  by  eliminating  observations  that  fall  outside  the  region  of 
common  support. 

2.  Stage  Two 

Stage  Two  is  simply  conducting  a  normal  regression  analysis  using  the  matched 
sample.  To  form  unbiased  and  efficient  estimates,  I  use  the  inverse  of  propensity  scores 
generated  from  the  first  stage  as  weights  in  the  second  stage  regressions.  Four  outcome 
variables  are  evaluated  to  ascertain  the  impact  of  DL.  TQPR  is  the  only  continuous 
outcome,  while  Graduated,  Promoted,  and  Separated  are  all  binary  dependent  variables. 
An  original  least  squares  (OLS)  regression  is  an  obvious  choice  for  TQPR,  however  the 
binary  dependent  variables  require  some  thought. 

I  initially  specify  the  three  binary  outcomes  as  logit  models.  Upon  closer 
examination,  I  realize  that  a  Linear  Probability  Model  (LPM)  is  the  superior  choice.  The 
logit  model  derivatives,  or  marginal  effects,  for  all  binary  outcomes  are  nearly  identical 
to  the  coefficients  of  a  similarly  specified  LPM.  Further,  the  LPM’s  key  weakness  lies  it 
its  ability  to  produce  predicted  values  less  than  zero  or  greater  than  one,  which  are  not 
feasible  probability  values  (Woolridge,  2013).  I  generate  predicted  values  for  a  LPM 
regression  of  each  binary  outcome  and  summarize  the  results.  For  each  outcome,  the 
predicted  values  fall  within  the  acceptable  range  between  the  5th  and  95th  percentiles. 
Also  similar  to  its  OLS  counterpart,  the  LPM  is  the  best  linear  unbiased  estimator.  Thus, 
the  LPM  is  both  qualitatively  and  quantitatively  the  superior  choice  of  model.  Equations 
2  and  3  depict  the  specifications  for  continuous  and  binary  outcomes,  respectively. 

y,  =b0  +  bJ)L  +  b1DEMOi  +  b,ACADi  +  b4MIL  +  b5CHRT  (2) 

prob(yj  =  1  )  =  b0+  bfiL.  +  b1DEMOi  +  b3ACAD .  +  b4MIL.  +  b5CHRT  (3) 
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Due  to  the  matching  process,  bj  represents  the  treatment  effect  of  DL  in  both 
Equations  2  and  3.  Under  propensity  score  matching  assumptions,  this  treatment  effect  is 
also  free  of  self- selection  bias. 

The  final  step  involves  running  several  regressions  to  determine  the  heterogeneity 
of  DL’s  effect  within  different  subgroups  of  the  matched  sample.  I  run  a  series  of  112 
separate  regressions  where  each  service,  community,  rank,  APC,  sector,  and  school  are 
isolated.  Equations  4  and  5  depict  the  specifications  for  continuous  and  binary  outcomes, 
respectively,  where  CONTROL;  indicates  the  subgroup  being  examined  in  isolation. 

y,  =b0+  biDEMOi ,  s .  t . CONTROL  .  =  1  (4) 

prob(yi  =  1  )  =  b0+  blDEMOi ,  s .  t  .CONTROL  j  =  1  (5) 

These  final  models  allow  comparison  of  DL’s  effect  between  different  services, 
communities,  ranks,  schools,  and  among  those  students  that  did  or  did  not  meet  the 
requisite  APC  for  their  curriculum. 
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IV.  RESULTS 


In  this  chapter,  I  present  the  results  in  three  sections.  First,  I  show  the  first  stage 
results  of  the  matching  exercise.  Second,  I  present  the  second  stage  results  estimating  the 
impact  of  DL  on  the  four  outcome  variables,  namely  TQPR,  Graduate,  Promoted,  and 
Separated.  Finally,  I  test  for  heterogeneous  effects  of  DL  within  subgroups  of  the 
matched  sample. 

A.  STAGE  ONE 

As  mentioned  in  Chapter  III,  to  create  a  matched  sample  I  first  regress  observable 
characteristics  of  students  on  an  indicator  for  DL  participation  using  logistic  regressions. 
The  key  observable  factors  included  in  the  logit  model  are  demographic  characteristics, 
military  service  branch,  warfare  community  and  its  associated  interactions  with  service 
branch,  and  military  rank.  The  full  results  of  this  regression  appear  in  Appendix  A.  It  is 
important  to  note  that  gender,  marital  status,  time  elapsed  since  undergraduate  education, 
undergraduate  university  sector,  service  branch,  the  support  community,  and  support 
community  and  service  interactions  are  all  significant  predictors  of  enrolling  in  DL.  Also, 
the  ranks  of  0-2  and  0-3  are  significant  predictors  of  selection  of  DL  programs  as  well. 

From  the  logistic  regression,  I  calculate  predicted  values  for  the  likelihood  of  DL 
enrollment  and  then  plot  probability  density  overlays  for  different  combinations  of 
observable  characteristics.  Figure  13  shows  the  probability  density  functions  separately 
for  resident  and  distance  learning  students. 
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Figure  13.  Propensity  score  matching  overlay 


Pr(dl) 

Residence  Learning  -  Distance  Learning 


The  x-axis  of  the  plot  in  Figure  13  represents  the  propensity  score.  These  values 
represent  the  predicted  values  resulting  from  the  logistic  regression.  The  region  of 
common  support  exists  between  approximately  0.1  and  0.3.  Observed  students  falling 
outside  of  this  range  are  eliminated  from  the  sample  as  they  are  significantly  dissimilar  to 
those  within  the  region  of  common  support.  Doing  so  results  in  a  matched  sample  of 
5,289  observations.  This  reduction  of  only  1,465  students  from  the  initial  sample  results 
in  a  matched  sample  that  is  sufficiently  large  and  robust  to  provide  statistical  power  to 
my  analysis. 

B.  STAGE  TWO 

Table  5  displays  the  average  treatment  effect  (ATE)  of  DL  for  each  performance 
outcome.  These  results  are  representative  of  the  entire  matched  sample.  Also,  full 
regression  results  appear  in  Appendices  B  through  E. 
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Table  5.  Average  treatment  effect  summary 


Outcome 

ATE  of  DL 

TQPR 

-0.3998*** 

[0.0735] 

Graduate 

-0.2232*** 

[0.0272] 

Promoted 

-0.1253*** 

[0.0276] 

Separated 

0.0562*** 

[0.0203] 

Robust  standard  errors  in  brackets 
***  p<0.01,  **  p<0.05,  *  p<0.1 


The  first  row  of  Table  5  focuses  on  the  results  for  TQPR  (average  GPA).  All  else 
equal,  the  TQPR  of  students  in  DL  programs  is  0.3998  points  below  resident  students. 
This  result  is  statistically  significant  at  the  1  percent  level.  This  negative  impact  translates 
into  almost  half  a  letter  grade  reduction  in  a  DL  student’s  TQPR  compared  to  an 
equivalent  resident  student. 

It  is  also  important  to  note  that  DL  is  not  the  only  factor  that  affects  TQPR. 
Appendix  B  displays  complete  regression  results  for  this  outcome.  Married  students  on 
average  have  higher  TQPRs  than  unmarried  students.  Marine  Corps  students  have 
approximately  a  half  a  letter  grade  higher  TQPR  than  Navy  students.  Surface  warfare 
officers  and  submariners  both  have  lower  TQPRs  than  support  officers,  while  Naval 
Aviators  have  higher  TQPRs.  Students  in  GSEAS,  GSOIS,  and  SIGS  all  have  lower 
TQPRs  than  students  in  GSBPP.  Finally,  service  academy  graduates  have  significantly 
higher  TQPRs  than  students  that  graduated  from  public  institutions  for  their 
undergraduate  degree.  Interestingly,  meeting  APC  3  results  in  a  0.2127  point  reduction  in 
TQPR  on  average. 

Based  on  the  existing  literature,  it  appears  that  NPS  students  enrolled  in  DL 

programs  are  significantly  worse  off  than  their  civilian  equivalents.  These  results  are 

somewhat  inconsistent  with  the  common  overall  findings  of  little  to  no  significant 

difference  between  traditional  and  online  class  formats.  However,  they  are  consistent 

with  the  findings  of  Bernard  et  al.  (2004)  in  that  synchronous  distance  programs  perform 
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worse  than  asynchronous  equivalents.  The  NPS  DL  programs  being  synchronous  support 
the  general  finding  of  lower  performance  in  the  synchronous  format  compared  to 
asynchronous. 

The  second  row  of  Table  5  shows  the  results  for  graduation.  All  else  being  equal, 
I  find  that  DL  results  in  a  decrease  in  the  probability  of  graduating  by  22.32  percentage 
points  on  average.  This  negative  impact  is  significant  at  the  1  percent  level.  This  result 
implies  that  even  if  resident  programs  achieved  a  100  percent  graduation  rate,  that  DL 
equivalents  would  suffer  more  than  one-fifth  of  their  students  failing  to  complete 
programs  of  study.  Again,  synchronous  DL  at  NPS  seems  more  detrimental  than  existing 
literature  would  suggest. 

Similar  to  TQPR,  other  factors  significantly  affect  the  chances  of  graduating  from 
NPS.  For  example,  Appendix  C  shows  that  on  average  black  students  are  11  percentage 
points  less  likely  to  graduate  than  white  students.  Also  like  TQPR,  being  a  surface 
warfare  officer  or  a  submariner  results  in  a  negative  impact  on  the  probability  of 
graduation.  Marine  Corps  support  officers  and  Air  Force  support  officers  are  also  less 
likely  to  graduate.  Further,  0-5s  are  less  likely  to  graduate  than  0-3s  and  students  in 
SIGS  are  not  significantly  less  likely  to  graduate  than  GSBPP  students.  Again,  meeting 
APC  3  has  a  negative  impact  on  the  probability  of  graduation.  However,  APC  2  shows  a 
7  percentage  point  increase  in  the  probability  of  graduating. 

The  third  row  of  Table  5  suggests  DL  students  are  less  likely  to  promote  after 
NPS.  The  coefficient  on  DL  indicates  they  are  less  likely  to  promote  by  12.53  percentage 
points  on  average.  This  negative  impact  is  significant  at  the  1  percent  level.  Considering 
that  many  military  officers  view  graduate  education  as  a  means  of  increasing  their  chance 
of  promotion,  this  particular  effect  of  DL  is  noteworthy. 

Unlike  TQPR  and  graduation,  the  impact  of  DL  on  promotion  is  smaller  than 
several  other  effects.  Appendix  D  shows  that  NPS  Marine  Corps  officers  are 
20  percentage  points  more  likely  to  promote  than  Navy  officers  at  NPS.  Surface  warfare 
officers  and  Aviators  are  significantly  less  likely  to  promote  than  support  officers.  Also, 
0-4s,  0-5s,  and  0-6s  are  all  significantly  less  likely  to  promote  after  NPS  than  0-3s. 
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The  fourth  row  of  Table  5  shows  the  impact  of  DL  on  separation.  All  else  equal,  I 
find  that  DL  results  in  an  increase  in  the  probability  of  separating  from  the  military  by 
5.62  percentage  points  on  average.  This  result  is  significant  at  the  1  percent  level. 
Although  a  positive  value,  separation  represents  a  negative  outcome,  thus  DL  provides 
yet  another  negative  impact  on  performance  at  NPS.  Although  the  percentage  is  small, 
this  negative  impact  of  DL  suggests  that  the  military  would  have  a  more  difficult  time 
obtaining  a  return  on  investment  for  DL  graduates. 

Appendix  E  shows  the  other  factors  affecting  the  probability  of  separating  from 
active  service.  Similar  to  promotion  results,  DL  is  one  of  the  smaller  significant  effects 
on  separation.  Being  Hispanic  results  in  a  6  percentage  point  decrease  in  the  probability 
of  separating  after  NPS,  while  being  married  results  in  an  1 1  percentage  point  decrease  in 
the  probability  of  separation  after  NPS.  Marine  Corps  officers  are  also  11  percentage 
points  less  likely  to  separate  than  Navy  officers.  Students  graduating  from  Private  for- 
profit  institutions  for  their  undergraduate  degrees  are  also  12  percentage  points  less  likely 
to  separate.  Although  DL  is  small  in  magnitude,  it  is  one  of  the  only  characteristics  that 
increase  the  likelihood  of  separation  after  NPS. 

C.  HETEROGENEITY 

An  important  finding  across  several  studies  in  the  literature  is  the  presence  of 
heterogeneous  effects  of  distance  or  online  instruction  on  student  characteristics.  In 
section  B,  I  estimate  an  average  treatment  effect  for  all  types  of  students.  In  this  section,  I 
split  the  sample  into  various  subgroups  to  test  for  treatment  heterogeneity.  Specifically  I 
split  the  sample  by  service  branch,  warfare  community,  rank,  APC,  sector,  and  school  in 
the  following  sections.  Each  of  the  following  sections  represents  groupings  of  these 
mutually  exclusive  subgroups.  For  example,  individual  services  are  mutually  exclusive, 
as  a  student  cannot  be  in  both  the  Navy  and  Army  simultaneously.  A  student  is  either  in 
the  Navy,  the  Army,  The  Marine  Corps,  or  the  Air  Force. 

Appendices  F  through  I  show  full  regression  results  for  the  outcomes  TQPR, 
Graduate,  Promoted,  and  Separated,  respectively.  Within  these  appended  tables  each  row 
represents  an  individual  regression  for  the  sample  subgroup  listed  in  the  first  column. 
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Tables  6  through  11  compile  the  coefficients  for  DL  within  each  of  the  four  appended 
tables  of  full  results.  Also,  all  tables  within  this  section  highlight  sample  subgroups  in  red 
to  represent  small  sample  sizes  of  less  than  100  students  enrolled  in  DL.  These 
highlighted  subgroups  indicate  results  that  are  considered  insignificant  as  they  lack 
sufficient  statistical  power  to  determine  the  effect  of  DL. 

1.  Service 

Table  6  represents  the  treatment  effect  of  DL  within  each  of  the  four  services 
present  within  the  matched  sample.  Within  the  Marine  Corps  and  Army  no  significant 
difference  exists  between  DL  and  resident  students  with  respect  to  TQPR.  However, 
Marine  DL  students  tend  to  have  lower  probabilities  of  graduating  than  Marine  residents. 
Army  DL  students  tend  to  have  lower  graduation  rates,  lower  promotion  rates,  and  higher 
separation  rates  than  residents. 


Table  6.  Treatment  heterogeneity  by  service 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

Navy 

-0.1647*** 

-0.0778*** 

0.0477** 

-0.0020 

3,055 

776 

MarineCorps 

-0.2616 

-0.2122** 

-0.0792 

0.0410 

684 

41 

Army 

-0.8085 

-0.2611 

-0.3940*** 

0.2113*** 

755 

7 

AirForce 

-0.1833*** 

-0.1644*** 

-0.3997*** 

0.2012*** 

754 

55 

Navy  DL  students  have  significantly  decreased  TQPRs,  a  lower  probability  of 
graduation,  and  a  higher  probability  of  promotion  after  NPS  than  resident  Navy  students. 
DL  has  no  significant  effect  on  separation  within  the  Navy  as  a  whole.  Marine  Corps, 
Army,  and  Air  Force  students  have  no  significant  effect  due  to  insufficient  sample  size. 

2.  Community 

Table  7  represents  the  treatment  effect  of  DL  within  different  warfare 
communities.  Due  to  sample  size  limitations,  only  the  aviation  and  support  communities 
are  considered  significant.  Aviators,  a  numerically  important  group,  show  smaller 
negative  effects  of  DL  on  TQPR  with  no  significant  impact  of  DL  on  the  probability  of 
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promotion  or  separation.  DL  students  within  the  support  community  show  similar  effects 
of  DL  to  the  overall  sample.  The  exception  is  for  separation,  as  students  in  the  support 
community  show  no  significant  impact  of  DL  on  separation  from  active  service. 


Table  7.  Treatment  heterogeneity  by  community 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

Surface 

-0.5322*** 

-0.2837*** 

-0.2226*** 

-0.0430 

695 

68 

Submarine 

-1.2189*** 

-0.5338*** 

-0.2869*** 

-0.0749*** 

230 

51 

Aviation 

-0.0646*** 

-0.0014 

-0.0361 

0.0315 

1,164 

512 

Intelligence 

-0.4580 

-0.1277 

-0.1902*** 

0.3359* 

590 

13 

GroundComb; 

-0.4520** 

-0.2332** 

-0.2917*** 

0.3681*** 

684 

18 

Support 

-0.3375** 

-0.1997*** 

-0.2024*** 

0.0452 

2,131 

207 

3.  Rank 

The  varying  treatment  effects  of  DL  with  respect  to  individual  ranks  are  shown  in 
Table  8.  0-3s  and  0-5s  are  the  only  ranks  with  a  significant  negative  impact  of  DL  on 
TQPR,  with  0-5 ’s  showing  slightly  more  negative  results.  With  respect  to  graduation,  O- 
5s  again  see  the  largest  decrease  in  probability  for  DL.  0-4s  are  the  only  rank  to  see  a 
significant  impact  on  promotion  between  DL  and  residency,  which  negative  and  almost 
twice  as  much  so  as  the  full  sample  average.  Finally,  DL  0-3s  are  significantly  more 
likely  to  separate  than  their  resident  counterparts. 


Table  8.  Treatment  heterogeneity  by  rank 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

01 

-1.0511 

-0.2817 

-0.2514 

-0.0078 

123 

13 

02 

-0.1901* 

-0.0873 

0.0038 

0.1515** 

438 

29 

03 

-0.1430*** 

-0.1014*** 

-0.0469 

0.0605** 

2,950 

530 

04 

-0.3173* 

-0.1545*** 

-0.2406*** 

0.0570 

1,604 

198 

05 

-0.2104** 

-0.3534*** 

0.0207 

-0.1503* 

139 

100 

06 

-0.1747 

-0.4017 

-0.0920 

0.0000 

11 

9 
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4. 


APC 


Table  9  illustrates  the  varying  effects  of  treatment  for  DL  with  respect  to 
academic  preparation.  There  is  a  clear  and  significant  difference  in  the  impact  of  DL 
between  students  that  did  and  did  not  meet  each  individual  APC  category. 


Table  9.  Treatment  heterogeneity  by  APC 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

met_apc  1 

-0.3015*** 

-0.1619*** 

-0.1086*** 

0.0462** 

611 

185 

met_apc2 

-0.2574*** 

-0.1200*** 

-0.1088*** 

0.0560** 

1,271 

278 

met_apc3 

-0.2961*** 

-0.1558*** 

-0.1101*** 

0.0486** 

551 

140 

notmet_apcl 

-0.0265 

-0.0695* 

-0.1202** 

0.1663*** 

4,768 

694 

notmet_apc2 

-0.1501 

-0.0813* 

-0.0416 

0.1265*** 

4,018 

601 

notmet apc3 

0.0199 

-0.0360 

-0.0655 

0.1468*** 

4,738 

739 

These  results  are  inconsistent  with  existing  literature  in  that  stronger  or  more 
prepared  students  tend  to  perform  worse  in  DL.  For  students  who  meet  APC 
requirements,  DL  has  a  significant  and  negative  impact  on  all  four  outcomes.  However, 
students  who  do  not  meet  the  requisite  APC  show  no  significant  difference  between  DL 
and  residency.  These  results  are  increasingly  relevant  when  considering  the  data 
breakdowns  presented  in  Chapter  III.  Not  only  do  DL  students  not  meeting  requisite  APC 
perform  no  worse  than  resident  students,  but  the  percentage  of  students  not  meeting  APC 
within  DL  is  roughly  twice  that  of  within  resident  programs. 

5.  Sector 

The  type  of  institution  students  receive  their  undergraduate  degrees  presents  more 
interesting  information.  Table  10  shows  the  treatment  effect  of  DL  for  the  four  sectors  of 
undergraduate  institutions  within  this  study.  Service  Academy  graduates  display  no 
significant  effect  of  DL.  Further,  students  who  graduated  from  public  civilian  universities 
for  their  undergraduate  degree  experience  a  significant  and  more  negative  impact  of  DL 
than  students  graduating  from  private  not-for-profit  institutions. 
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Table  10.  Treatment  heterogeneity  by  sector 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

Public 

-0.3541*** 

-0.1565*** 

-0.0880** 

0.0695** 

2,679 

363 

PrivateNFP 

-0.1921*** 

-0.1524*** 

-0.0666 

0.0418 

1,123 

186 

PrivateFP 

0.1093 

-0.1509 

-0.1796 

-0.0101 

43 

4 

Service  Acade 

-0.0727* 

-0.0591* 

-0.1456*** 

0.1432*** 

1,435 

324 

It  is  also  interesting  to  see  that  Service  Academy  graduates  experience  the  greatest 
decrease  in  probability  of  promotion  and  the  largest  increase  in  probability  of  separation 
as  a  result  of  DL.  Students  from  public  civilian  institutions  have  a  similar  impact  of  DL 
on  promotion  and  separation;  however,  the  magnitude  is  roughly  half  that  of  Service 
Academy  graduates. 

6.  School 

Schools  within  NPS  also  present  some  interesting  heterogeneity.  Table  11  shows 
treatment  effects  of  DL  for  the  four  schools.  GSEAS  and  GSOIS  show  similarly  negative 
impacts  for  DL  with  respect  to  both  TQPR  and  probability  of  graduation.  They  differ 
with  respect  to  late  graduation,  however.  DL  actually  has  a  significant  and  slightly 
positive  effect  on  the  probability  of  graduating  on  time  within  GSOIS.  More  relevant, 
however,  is  that  students  within  GSBPP  show  no  significant  difference  between  DL  and 
residency  on  all  outcomes  except  the  probability  of  graduation. 


Table  11.  Treatment  heterogeneity  by  school 


ATE  of  DL 


TQPR 

Graduate=l 

Promoted 

Separated 

n 

DL=1 

GSBPP 

-0.0982 

-0.1114*** 

-0.0424 

0.0264 

1,477 

555 

GSEAS 

-0.2147*** 

-0.1107*** 

-0.0508 

0.0760* 

1,099 

203 

GSOIS 

-0.7181*** 

-0.2646*** 

-0.1691*** 

0.0437 

1,509 

119 

SIGS 

0.0981** 

0.2404*** 

-0.4227*** 

0.7918*** 

1,086 

2 
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V.  SUMMARY  AND  CONCLUSIONS 


A.  SUMMARY 

I  merge  data  from  three  separate  sources  to  create  a  robust  sample  of  NPS  military 
officer  students.  These  data  contain  sound  and  diverse  demographic  and  undergraduate 
institution  information.  Using  these  diverse  controls  and  employing  propensity  score 
matching,  I  mitigate  the  pervasive  issues  present  within  the  existing  literature  addressing 
observational  studies,  small  sample  sizes,  and  demographic  controls. 

Utilizing  the  matched  sample  I  find  that  DL  has  a  significant  and  negative  impact 
on  both  student  academic  performance  and  subsequent  job  performance.  Based  on  the 
findings  of  existing  literature,  NPS  students  seem  to  be  worse  off  than  civilian 
equivalents.  Further,  results  are  heterogeneous  within  different  military  services,  warfare 
communities,  and  levels  of  academic  preparation. 

B.  RECOMMENDATIONS 

When  looking  at  the  results,  several  potential  means  of  improving  the  impact  of 
DL  come  to  mind.  First,  focusing  on  programs  or  students  who  are  more  likely  to 
perform  well  in  DL  could  help  mitigate  the  existing  negative  impact  on  performance.  For 
example,  students  within  GSBPP  show  no  significant  difference  in  performance  between 
DL  and  residency.  With  regard  to  warfare  communities,  Aviators  suffer  no  significant 
impact  of  DL  as  well.  However,  it  is  worth  noting  that  several  of  these  student 
characteristics  that  tend  to  favor  DL  are  already  heavily  represented  within  the  existing 
DL  population.  More  than  half  of  the  students  within  the  sample  are  already  enrolled  in 
programs  within  GSBPP.  Also,  more  than  half  of  all  DL  students  between  2006  and  2013 
are  Aviators.  Thus,  due  to  the  existing  makeup  of  DL,  this  solution  may  not  be  entirely 
feasible. 

Another  area  of  potential  improvement  revolves  around  APC  requirements.  I 

recommend  creating  less  stringent  requirements  for  admission  to  DL,  or  simply 

continuing  to  waive  existing  requirements.  Chapter  III  clearly  demonstrated  the  large 

number  of  students  within  DL  who  do  not  meet  requisite  APC  values  for  their 
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curriculums.  This  observation  is  even  more  relevant  when  coupled  with  the 
heterogeneous  findings  that  students  who  do  meet  requisite  APC  suffer  a  significantly 
negative  impact  of  DL  while  those  that  do  not  meet  requisite  APC  show  no  significant 
effect  of  DL.  Waiving  APC  requirements  within  DL  does  not  seem  to  be  manifesting  any 
significantly  negative  effect  on  students’  grades. 

Next,  I  recommend  shifting  DL  at  NPS  to  a  more  asynchronous  format.  Although 
there  are  asynchronous  aspects  of  classes  within  NPS  DL  programs,  they  are  primarily 
synchronous  in  their  delivery.  The  existing  literature  clearly  addresses  the  superiority  of 
asynchronous  formats  over  synchronous  formats  with  regard  to  DL.  Further,  as  most  of 
the  studies  that  find  no  significant  difference  between  distance  and  traditional  formats  are 
based  on  asynchronous  online  programs,  this  difference  in  format  could  help  explain  the 
uncharacteristically  negative  impact  of  DL  at  NPS. 

Finally,  I  recommend  conducting  additional  research  on  the  impact  of  DL  at  NPS. 
Particularly,  the  ability  to  control  for  individual  course  grades  seems  a  logical  next  step. 
The  majority  of  existing  literature  focusses  on  individual  courses  and  their  associated 
course  grades  when  determining  the  impact  of  DL.  As  such,  utilizing  similar  performance 
outcomes  could  provide  a  better  comparison  to  civilian  equivalents. 
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APPENDIX  A.  PROPENSITY  SCORE  MATCHING-LOGIT 

RESULTS 


Matching  Logit  -  DL 

VARIABLE  COEFFICIENT 


Female 

-0.3603** 

[0.1710] 

Black 

-0.2683 

[0.1754] 

Hispanic 

-0.0043 

[0.1636] 

OtherRace 

-0.0314 

[0.1461] 

Married 

0.5472*** 

[0.1311] 

Divorced 

0.0837 

[0.5625] 

Age 

-0.0051 

[0.0148] 

YrsFrUGrad 

0.0750*** 

[0.0153] 

DEPS 

-0.0543 

[0.0369] 

sector 

-0.1992*** 

[0.0587] 

MarineCorps 

-1.5558*** 

[0.2047] 

Army 

-4.4654*** 

[0.5668] 

AirForce 

-2.4622*** 

[0.3605] 

Support 

1.1198*** 

[0.3806] 

Navy_Support 

-2.0320*** 

[0.3959] 

Marine_Support 

-1.8838*** 

[0.5066] 

Army  Support 

-0.2678 

[0.8399] 

Ol 

-0.1529 

[0.3314] 

02 

-0.7505*** 

[0.2095] 

04 

-0.1091 

[0.1261] 

05 

1.8117*** 

[0.2802] 

06 

1.8150** 

[0.9055] 

Constant 

-1.3519*** 

[0.3987] 

Observations 

5,290 

Robust  standard  errors  in  brackets 

***  p<0.01, 

**  p<0.05. 

*  p<0.1 
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APPENDIX  B.  OLS  REGRESSION  RESULTS  FOR  THE  OUTCOME 

TQPR 


_ TQPR _ 

VARIABLE  COEFFICIENT  VARIABLE  COEFFICIENT 


dl 

-0.3998*** 

Marine_Support 

-0.6372*** 

[0.0735] 

[0.2242] 

Female 

-0.0879 

Army  Support 

-0.4874 

[0.0832] 

[0.3655] 

Black 

-0.1185 

AirForce_Support 

-0.1668* 

[0.0941] 

[0.0922] 

Hispanic 

-0.0089 

Ol 

-0.5124 

[0.0425] 

[0.3189] 

OtherRace 

-0.1616** 

02 

-0.0173 

[0.0747] 

[0.0733] 

Married 

0.1532*** 

04 

0.0063 

[0.0503] 

[0.0733] 

Divorced 

0.2381** 

05 

0.0376 

[0.1010] 

[0.0939] 

Age 

-0.0155 

06 

0.2965* 

[0.0106] 

[0.1679] 

DEPS 

-0.0208 

met_apcl 

-0.1146 

[0.0169] 

[0.0775] 

YrsFrUGrad 

0.0071 

met_apc2 

0.0831 

[0.0128] 

[0.1070] 

MarineCorps 

0.4011*** 

met_apc3 

-0.2127** 

[0.1088] 

[0.0836] 

Army 

0.0519 

ServiceAcademy 

0.1491*** 

[0.1499] 

[0.0519] 

AirForce 

0.0824 

PrivateNFP 

0.0481 

[0.0838] 

[0.0585] 

Surface 

-0.3383*** 

PrivateFP 

0.0561 

[0.0744] 

[0.0742] 

Submarine 

-0.4216*** 

GSEAS 

-0.1760** 

[0.1178] 

[0.0809] 

Aviation 

-0.2784* 

GSOIS 

-0.2681*** 

[0.1522] 

[0.0958] 

Intelligence 

0.1317 

SIGS 

-0.1286** 

[0.1710] 

[0.0645] 

GroundCombat 

-0.0967 

_Y  Start  Acad_2007 

0.1820 

[0.0702] 

[0.1243] 

Naval_Aviation 

0.3649** 

_Y  Start  Acad_2008 

0.2367** 

[0.1497] 

[0.1182] 

Marine_Aviation 

0.0369 

_YStart  Acad_2009 

0.2836** 

[0.1754] 

[0.1311] 

Army_Aviation 

0.0836 

_Y  Start  Acad_20 1 0 

0.0439 

[0.1957] 

[0.1555] 

AirForce_Aviation 

0.2127 

_Y  Start  Acad_20 1 1 

0.2971** 

[0.1592] 

[0.1405] 

Navy_Intelligence 

-0.1799 

_Y  Start  Acad_20 1 2 

0.2636** 

[0.1855] 

[0.1309] 

Marine_Intelligence 

-0.5109** 

_YStartAcad_20 1 3 

0.2170* 

[0.2178] 

[0.1257] 

Annv  Intelligence 

-0.1358 

QStartQuar  2 

0.0213 

[0.1791] 

[0.0662] 

Navy_GroundCombat 

-0.1763 

_QStartQuar_3 

-0.0386 

[0.1396] 

[0.0748] 

Marine_GroundCombat 

-0.4583*** 

_QStartQuar_4 

0.0616 

[0.1556] 

[0.0853] 

Army_GroundCombat 

0.0465 

Constant 

4.3355*** 

[0.1349] 

]0.2906] 

Marine_Support 

-0.6372*** 

[0.2242] 

Observations 

5,289 

Adjusted  R-squared 

0.190 

Robust  standard  errors  in  brackets 
***  p<0.01,  **  p<0.05,  *  p<0.1 
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APPENDIX  C.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 

GRADUATE 


GRADUATE 


VARIABLE 

COEFFICIENT 

VARIABLE  COEFFICIENT 

dl 

-0.2232*** 

Marine_Support 

-0.2033** 

[0.0272] 

[0.1034] 

Female 

-0.0402 

Army_Support 

0.0775 

[0.0395] 

[0.1547] 

Black 

-0.1105** 

[0.0508] 

AirForce_Support 

-0.2313*** 

[0.0553] 

Hispanic 

-0.0877 

Ol 

0.0037 

[0.0570] 

[0.0949] 

OtherRace 

-0.0490 

02 

0.0617 

[0.0327] 

[0.0421] 

Married 

0.0461 

04 

-0.0071 

[0.0310] 

[0.0276] 

Divorced 

0.0540 

05 

-0.1396*** 

[0.0494] 

[0.0534] 

Age 

-0.0039 

06 

0.1984 

[0.0035] 

[0.1250] 

DEPS 

-0.0116 

met_apcl 

-0.0613* 

[0.0086] 

[0.0336] 

YrsFrUGrad 

0.0005 

met_apc2 

0.0688** 

[0.0038] 

[0.0345] 

MarineCorps 

0.0630 

met_apc3 

-0.0861** 

[0.0884] 

[0.0373] 

Army 

-0.1442 

ServiceAcademy 

0.0331 

[0.1253] 

[0.0204] 

AirForce 

0.0850* 

PrivateNFP 

-0.0052 

[0.0465] 

[0.0261] 

Surface 

-0.1457*** 

PrivateFP 

-0.0906 

[0.0296] 

[0.1429] 

Submarine 

-0.2015*** 

GSEAS 

-0.1128*** 

[0.0370] 

[0.0296] 

Aviation 

0.0487 

GSOIS 

-0.0981*** 

[0.0803] 

[0.0298] 

Intelligence 

-0.0344 

SIGS 

-0.0446 

[0.0722] 

[0.0277] 

GroundCombat 

-0.0621 

_Y  S  tart  Acad_2007 

-0.0039 

[0.0473] 

[0.0369] 

Naval_Aviation 

-0.0465 

_YStartAcad_2008 

0.0199 

[0.0826] 

[0.0353] 

Maiine_Aviation 

-0.0563 

YStartAcad  2009 

0.0301 

[0.1189] 

[0.0414] 

Army_Aviation 

-0.0966 

YStartAcad  2010 

-0.0384 

[0.1303] 

[0.0432] 

AirForce_Aviation 

-0.1370* 

_Y  Start  Acad_20 1 1 

-0.0026 

Naw_Intelligence 

[0.0804] 

-0.0081 

YStartAcad  2012 

[0.0414] 

-0.0903** 

[0.0793] 

[0.0403] 

Marine_Intelligence 

-0.0172 

YStartAcad  2013 

-0.6134*** 

[0.1304] 

[0.0383] 

Armv_Intelligence 

0.0831 

_QStartQuar_2 

-0.0254 

[0.0908] 

[0.0268] 

N  a  vy_GroundC  ombat 

-0.0194 

_QStartQuar_3 

-0.0831*** 

[0.0612] 

[0.0311] 

Manne_GroundCombat 

-0.1658 

_QStartQuar_4 

-0.1225*** 

[0.1376] 

[0.0298] 

Army_GroundC  ombat 

0.2331** 

Constant 

1.3504*** 

[0.1181] 

[0.1157] 

Observations 

5,289 

Adjusted  R-squared 

0.364 

Robust  standard  errors  in  brackets 


***  p<0.01,  **  p<0.05,  *  p<0.1 
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APPENDIX  D.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 

PROMOTED 


_ PROMOTED _ 

VARIABLE  COEFFICIENT  VARIABLE  COEFFICIENT 


dl 

-0.1253*** 

Marine_Support 

-0.3884*** 

[0.0276] 

[0.0769] 

Female 

-0.0157 

Army_Support 

-0.0272 

[0.0484] 

[0.0788] 

Black 

-0.0408 

AirForce_Support 

-0.2065 

[0.0374] 

[0.1262] 

Hispanic 

-0.0138 

01 

0.2753*** 

[0.0379] 

[0.0958] 

OtherRace 

-0.0435 

02 

-0.1268*** 

[0.0292] 

[0.0390] 

Married 

0.0846*** 

04 

-0.0873*** 

[0.0307] 

[0.0287] 

Divorced 

0.0514 

05 

-0.4005*** 

[0.0698] 

[0.0460] 

Age 

-0.0101*** 

06 

-0.2171** 

[0.0036] 

[0.0964] 

DEPS 

-0.0104 

met_apcl 

0.0058 

[0.0085] 

[0.0407] 

YrsFrUGrad 

0.0018 

met_apc2 

0.0233 

[0.0041] 

[0.0255] 

MarineCorps 

0.2038*** 

met_apc3 

-0.0111 

[0.0647] 

[0.0396] 

Army 

-0.1308* 

ServiceAcademy 

0.0022 

[0.0683] 

[0.0223] 

AirForce 

0.0739 

PrivateNFP 

0.0016 

[0.1270] 

[0.0263] 

Surface 

-0.2172*** 

PrivateFP 

0.0727 

[0.0302] 

[0.0660] 

Submarine 

-0.0764* 

GSEAS 

-0.0628** 

[0.0404] 

[0.0296] 

Aviation 

-0.9628*** 

GSOIS 

-0.0736*** 

[0.2008] 

[0.0245] 

Intelligence 

-0.1469 

SIGS 

0.0349 

[0.1739] 

[0.0236] 

GroundCombat 

0.1565 

_Y  Start  Acad_2007 

-0.0345 

[0.1832] 

[0.0450] 

Naval_Aviation 

0.8828*** 

_Y  Start  Acad_2008 

-0.0930** 

[0.2001] 

[0.0416] 

Marine_Aviation 

0.6711*** 

_YStartAcad_2009 

-0.2989*** 

[0.2160] 

[0.0425] 

Army_Aviation 

0.9687*** 

_Y  Start  Acad_20 1 0 

-0.3362*** 

[0.2136] 

[0.0431] 

AirForce_Aviation 

1.1520*** 

_Y  Start  Acad_20 1 1 

-0.4546*** 

[0.2398] 

[0.0365] 

Navy  Intelligence 

-0.0653 

YStartAcad  2012 

-0.5902*** 

[0.1778] 

[0.0340] 

Marine_Intelligence 

-0.2166 

_Y  Start  Acad_20 1 3 

-0.6678*** 

[0.1906] 

[0.0318] 

Armv_Intelligence 

0.1968 

_QStartOuar_2 

-0.0675*** 

[0.1782] 

[0.0255] 

N  a  vy_GroundC  ombat 

-0.1288 

_OStartOuar_3 

-0.0176 

[0.1880] 

[0.0277] 

Marine_GroundCombat 

-0.5448*** 

_QStartQuar_4 

-0.1001*** 

[0.1983] 

[0.0256] 

Army_GroundCombat 

-0.1518 

Constant 

1.2456*** 

[0.1930] 

[0.1198] 

Observations 

5,289 

Adjusted  R-squared 

0.358 

Robust  standard  errors  in  brackets 

***  p<0.01, 

**  p<0.05,  *  p<0.1 
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APPENDIX  E.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 

SEPARATED 


SEPARATED 


VARIABLES 

COEFFICIENT 

VARIABLE 

COEFFICIENT 

dl 

0.0562*** 

Marine_Support 

0.3069*** 

[0.0203] 

[0.0723] 

Female 

-0.0214 

Army_Support 

-0.2274** 

[0.0323] 

[0.1072] 

Black 

-0.0069 

AirForce_S  upport 

0.0162 

[0.0258] 

[0.1387] 

Hispanic 

-0.0648** 

01 

-0.1167*** 

[0.0317] 

[0.0371] 

OtherRace 

0.0607* 

02 

0.1238*** 

[0.0359] 

[0.0433] 

Married 

-0.1100*** 

04 

0.0045 

[0.0264] 

[0.0260] 

Divorced 

-0.0852 

05 

0.1098* 

[0.0808] 

[0.0599] 

Age 

0.0063* 

06 

0.0051 

[0.0035] 

[0.0663] 

DEPS 

0.0092 

met_apcl 

0.0181 

[0.0076] 

[0.0428] 

YrsFrUGrad 

-0.0068* 

met_apc2 

-0.0272 

[0.0040] 

[0.0237] 

MarineCorps 

-0.1182*** 

met_apc3 

-0.0699** 

[0.0421] 

[0.0347] 

Army 

0.2211** 

ServiceAcademy 

0.0365 

[0.1083] 

[0.0225] 

AirForce 

0.0397 

PrivateNFP 

-0.0305 

[0.13811 

[0.0251] 

Surface 

0.0182 

PrivateFP 

-0.1167*** 

[0.0222] 

[0.03871 

Submarine 

0.0377 

GSEAS 

-0.0386 

[0.0247] 

[0.0257] 

Aviation 

0.0943 

GSOIS 

-0.0098 

[0.1037] 

[0.0255] 

Intelligence 

0.3509 

SIGS 

0.0414 

[0.2547] 

[0.0263] 

GroundCombat 

-0.1320*** 

_  Y  Start  Acad_2007 

-0.0091 

[0.0448] 

[0.0408] 

N  aval_Aviation 

-0.0109 

YStartAcad  2008 

0.0165 

[0.1068] 

[0.0388] 

Marine_Aviation 

0.2246* 

_Y  Start  Acad_2009 

-0.0502 

[0.1253] 

[0.0369] 

Army_Aviation 

-0.3499*** 

_Y  Start  Acad_20 1 0 

-0.0241 

[0.1347] 

[0.0385] 

AirForce_Aviation 

-0.1695 

_Y  Start  Acad_20 1 1 

-0.0524 

[0.1809] 

[0.0358] 

N  avy_Intelligence 

-0.2964 

YStartAcad  2012 

-0.0196 

[0.2559] 

[0.0350] 

Marine_Intelligence 

-0.1722 

_Y  Start  Acad_20 1 3 

-0.1518*** 

[0.2543] 

[0.0278] 

Army_Intelligence 

-0.6392** 

_QStartQuar_2 

-0.0265 

[0.2642] 

[0.0202] 

N  a  vy_GroundC  ombat 

0.1064** 

_QStartQuar_3 

-0.0672** 

[0.0518] 

[0.0297] 

Marine_GroundCombat 

0.2748*** 

_QStartQuar_4 

-0.0517*** 

[0.0813] 

[0.0190] 

Army_Grou  ndC  ombat 

0.2897*** 

Constant 

0.0775 

[0.1049] 

[0.1175] 

Observations 

5,289 

Adjusted  R-squared 

0.368 

Robust  standard  errors  in  brackets 


***  p<0.01,  **  p<0.05,  *  p<0.1 
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APPENDIX  F.  OLS  REGRESSION  RESULTS  FOR  THE  OUTCOME 

TQPR— HETEROGENEITY 
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APPENDIX  G.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 
GRADUATE— HETEROGENEITY 


l 

U 


it 


55 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


56 


APPENDIX  H.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 
PROMOTED— HETEROGENEITY 
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APPENDIX  I.  LPM  REGRESSION  RESULTS  FOR  THE  OUTCOME 
SEPARATED— HETEROGENEITY 
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